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LONG-TERM  GOALS 

The  long-term  goal  of  this  RTP  project  is  to  provide  the  warfighter  with  superior  battlespace 
environmental  awareness  in  terms  of  high  fidelity  four-dimensional  (4D)  depiction  of  the  global 
atmospheric  states.  This  situational  awareness  is  a  key  aspect  of  information  superiority  in  the  DoD’s 
strategic  plan  to  ensure  battlespace  dominance  in  the  21st  century.  This  goal  is  to  be  accomplished  by 
providing  NOGAPS1  with  best  possible  initial  condition  through  the  use  of  next  generation  global 
atmospheric  4D  variational  (4DVAR)  data  assimilation  system,  NAVDAS-AR2. 

OBJECTIVES 

The  objective  of  this  project  is  to  construct  and  transition  a  4DVAR  global  atmospheric  data 
assimilation  system  for  NOGAPS  to  Fleet  Numerical  Meteorology  and  Oceanography  Center 
(FNMOC).  This  system,  NAVDAS-AR,  represents  the  first  operational,  weak  constraint,  4DVAR 
atmospheric  data  assimilation  system  in  the  world.  In  this  context,  “weak  constraint”  means  that  the 
atmospheric  forecast  model  is  not  considered  a  “perfect”  model,  but  rather  is  assumed  to  have  errors. 
This  enables  the  most  optimal  solution.  NAVDAS-AR  will  provide  high  fidelity,  dynamically 
consistent  analyses  for  NWP  model  initialization  and  for  warfighter  support,  and  will  be  capable  of 
efficiently  handling  large  numbers  of  observations  that  may  be  irregularly  distributed  in  space  and 
time,  and/or  indirectly  related  to  the  model  state  variables  (e.g.,  satellite  radiances  or  wind  vectors). 

APPROACH 

Our  approach  is  to  build  on  the  prototype  of  NAVDAS-AR  (Xu  et  al  2005,  and  Rosmond  and  Xu 
2006)  that  has  been  developed  and  successfully  applied  to  the  global  4DVAR  data  assimilation 
application  using  the  NOGAPS  prediction  model  as  a  dynamic  constraint.  This  project,  which  is  a 
follow-up  to  a  NRL  ongoing  in-house  6.2  data  assimilation  project,  will  expand  this  prototype  to  a 
next-generation  operational  global  atmospheric  data  assimilation  system.  We  will  leverage  the 
existing  NAVDAS  and  NOGAPS  infrastructures  to  provide  the  pre-  and  post-  analysis  processes.  The 
system  will  be  thoroughly  tested  using  scientific  studies,  and  comprehensive  data  assimilation  and 
forecast  experiments.  Although  the  goals  are  ambitious,  they  are  realistic  because  the  theoretical  basis 
for  the  project  is  already  in  place  owing  to  great  progress  made  in  our  6.1  and  6.2  in-house  data 
assimilation  projects  on  variational  data  assimilation. 


1  NOGAPS:  Navy  Operational  Global  Atmospheric  Prediction  System 

2  NAVDAS-AR:  NRL  Atmospheric  Data  Assimilation  System  -  Accelerated  Representer 
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WORK  COMPLETED 


The  following  is  a  list  of  work  completed  related  to  this  project  during  FY07. 

1 .  Developed  and  tested  a  new  highly  parallel  2D  domain  decomposition  algorithm  that  greatly 
improved  the  runtime  speed  of  the  NAVDAS-AR. 

2.  Developed  the  adjoint  system  of  NAVDAS-AR,  which  is  capable  of  examining  the  temporal  aspect 
of  the  observation  sensitivities. 

3.  Implemented  and  tested  a  new  minimization  algorithm,  Flexible  Conjugate  Gradient  (FCG),  to 
replace  the  current  standard  Conjugate  Gradient  (CG). 

4.  Implemented  and  tested  a  convolution  algorithm,  which  accounts  for  the  impact  of  model  error  in 
4DVAR  algorithm  such  as  NAVDAS-AR,  to  the  Burger’s  equation. 

5.  Completed  the  development  and  test  of  AMSU-A  radiance  assimilation  and  the  associated  bias 
correction.  We  also  started  to  assimilate  temperature  and  humidity  information  from  other  microwave 
and  infrared  sensors. 

6.  Conducted  extensive  data  assimilation  experiments  for  July-Aug  2005  and  Feb-March  2006. 

7.  Started  to  run  NAVDAS-AR  in  real  time  pre-operational  mode  on  FNMOC’s  new  Linux  Cluster 
system  -  A2  and  the  IBM  -  AMS  5. 

These  FY07  accomplishments  are  critical  for  the  comprehensive  data  assimilation  tests  under  real  time 
operational  environment  and  the  AMP  transition  of  NAVDAS-AR  to  FNMOC  planned  for  FY08. 

RESULTS 

The  following  results  represent  some  highlights  of  the  several  significant  accomplishments  of  this 
project  during  FY07. 

A  new  parallel  computation  algorithm  for  NA  VDAS-AR 

One  of  the  important  milestones  set  for  FY07  was  to  substantially  improve  the  actual  runtime  of 
NAVDAS-AR.  A  new  2D  domain  decomposition  parallel  algorithm  has  been  developed  for 
NAVDAS-AR  by  Dr.  Rosmond.  The  original  NAVDAS-AR  code  was  written  for  only  a  ID  latitude 
band  decomposition,  which  limited  the  maximum  processor  number  to  60  for  the  current  T79  inner 
loop  resolution.  While  presently  that  is  not  a  limitation,  since  we  always  run  with  40  processors  on  the 
FNMOC’s  IBM-AMS5,  future  systems  may  allow  much  larger  processor  numbers.  If  we  want  to 
increase  the  resolution  of  the  inner  loop,  and  also  reduce  AR  runtime,  we  need  the  ability  to  run  with 
substantially  more  processors  if  available.  With  2D  decomposition  and  T79  we  can  run  with  up  to  450 
processors,  and  for  a  T159  inner  loop  (this  is  what  ECMWF  runs),  up  to  1800  processors. 
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An  added  benefit  of  the  2D  decomposition  is  that  there  is  a  noticeable  decrease  in  runtime  for  some  of 
the  now  allowable  IPROC/JPROC  combinations.  Attached  is  a  histogram  plot  of  AR  runtime  for 
several  of  these  combinations.  The  1x40  far  left  bar  is  the  current  ID  decomposition  (IPROC=l, 
JPROC=40).  The  4x10  and  5x8  times  are  about  12%  less  that  the  1x40.  Interestingly,  this  is  contrary 
to  our  experience  with  the  NOGAPS  forecast  model,  which  has  always  had  2D  decomposition.  Any 
IPROC=l,  JPROC=xx  has  almost  always  been  the  best  choice  for  the  model.  Apparently  ANAVDAS- 
AR  benefits  from  a  more  even  distribution  of  processors  due  to  better  load  balance  in  observation 
space  and  more  efficient  MPI  when  going  between  observation  volume  space  and  grid  point  lat-lon 
space. 


ARSWEEP  RUNTIMES 


IPROCxJPROC 


Figure  1:  The  new  2D  domain  decomposition  parallel  computation  algorithm  not 
only  allows  NAVDAS-AR  to  use  more  processors,  but  also  produces  a  noticeable 
decrease  in  runtime  for  some  of  the  now  allowable  IPROC/JPROC  combinations. 
[Graphic  shows  the  reduction  of  runtime  for  NA  VDAS-AR  using  the  new  parallel  algorithm.] 


The  Adjoint  system  of  NA  VDAS-AR 

The  adjoint  of  NRL  Atmospheric  Variational  Data  Assimilation  System  (NAVDAS),  an  observation- 
space  3D-Var,  has  been  developed  at  NRL  in  Monterey,  CA,  USA.  The  adjoint  of  NAVDAS  has 
proven  to  be  very  useful  in  estimating  the  impact  of  observations  on  the  short-range  forecast  error  in 
NWP  in  the  past  several  years.  Due  to  the  limitations  of  3D-VAR,  however,  the  NAVDAS  adjoint 
does  not  properly  account  for  the  temporal  distribution  of  the  observations.  To  address  the  temporal 
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aspects  of  observation  impact,  we  are  currently  developing  and  testing  the  adjoint  of  NAVDAS-AR 
(accelerated  representer),  an  observation-space  4D-Var,  at  NRL-Monterey.  The  new  capability  of 
estimating  the  temporal  aspect  of  observation  impact  is  likely  to  become  increasingly  important  in  the 
future  as  asynoptic  observations  become  much  more  dominant.  In  this  presentation,  we  first  give  a 
brief  description  of  the  formulation  and  basic  design.  Figure  2  shows  the  results  of  assessing  the 
observation  impact  using  the  adjoint  system  of  NAVDAS-AR.  The  observation  impact  is  sorted  by  the 
observation  types.  More  details  are  given  by  Xu  et  al.  (2006  and  2007). 
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Figure  2:  The  ability  of  the  adjoint  of  NA  VDAS-AR  to  assess  and  monitor  the  impact  of 
observations  on  NOGAPS  forecast.  [Graphic  shows  RA  OB  is  the  most  beneficial  observation  type 
that  improves  the  NOGAPS  forecast ,  followed  by  AMSU-A,  SATWIND,  AIRCRAFT \  etc.] 


A  new  minimization  algorithm  (FCG)  for  NA  VDAS-AR 

Conjugate  gradient  algorithm  has  been  widely  used  in  find  solutions  for  high-dimension  linear 
equations.  However,  the  convergence  rate  of  standard  conjugate  gradient  algorithm  decreases  after 
certain  number  of  iterations  due  to  the  inability  of  the  CG  algorithm  to  keep  the  current  and  previous 
search  directions  being  orthogonal  with  each  other.  This  issue  is  caused  by  the  computational  errors 
caused  by  the  finite  accuracy  of  computers.  The  new  FCG  algorithm  avoids  the  pitfall  of  the  CG  by 
reorthogonal  the  search  directions  such  that  the  orthogonality  of  the  search  directions  is  always 
preserved.  As  we  can  see  from  Fig  3,  the  FCG  reaches  the  so  called  “super  linear  convergence. 

An  efficient  weak  constraint  algorithm  to  be  used  in  NA  VDAS-AR 

Strong  constraint  (perfect  model  assumption)  4D-Var  algorithms  are  increasingly  used  for  synoptic 
and  global  scales  atmospheric  data  assimilation  at  operational  numerical  weather  prediction  centers 
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around  the  world.  The  data  assimilation  windows  currently  range  from  6  hours  to  12  hours  at  different 
centers.  It  is  preferable  to  have  as  many  independent  observations  as  possible  in  each  data  assimilation 
window  under  the  variational  framework.  Longer  data  assimilation  windows  generally  increase  the 
information  content  from  the  observations,  but  also  make  the  perfect  model  assumption  more 
improper.  It  is  clear  that  weak  constraint  (imperfect  model  assumption)  4D-Var  algorithms  will  be 
required  to  properly  combine  the  background  forecast  with  high  resolution  observations  in  longer  data 
assimilation  windows  in  the  not  too  distant  future. 

Despite  the  weak  constraint  formulation,  the  current  NAVDAS-AR  implementation  uses  the  perfect 
model  assumption.  We  proposed  to  treat  the  impact  of  the  model  errors  as  a  residue  term  resulted  from 
a  four-dimensional  matrix/vector  multiplication  of  the  model  error  covariance  with  the  adjoint  field  in 
the  right  hand  side  of  the  tangent  linear  model.  The  calculation  of  the  residue  term  is  generally  very 
computationally  intensive.  However,  it  can  be  obtained  accurately  and  efficiently  when  some  special 
choices  are  made  for  the  model  error  covariances  using  a  special  approach  (Chua  and  Bennett,  2001). 
The  steps  involved  in  this  approach  are  demonstrated  through  application  to  the  Burgers'  equation  (Xu 
et  al.  2007). 
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Figure  3:  The  number  of  iterations  required  for  NA  VDAS-AR  to  converge  to  the  specified  criteria 
(Omega(log)=0.01).  Same  color  represents  one  date-time-group  of  data  assimilation  cycle  with  solid 
lines  for  FCG  and  dotted  lines  for  CG.  [Graphic  shows  FCG  consistently  requires  less  number  of 
iterations  (less  computational  cost)  to  converge  than  the  CG  does.] 
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IMPACT/APPLICATIONS 


The  current  operational  data  assimilation  system  at  FNMOC,  NAVDAS,  is  based  on  a  three- 
dimensional  variational  (3DVAR)  algorithm  and  is  cast  in  observation  space.  The  3DVAR  algorithm 
is  widely  used  in  intermittent  cycling  data  assimilation  for  the  analysis  of  global  and  synoptic  scales 
around  the  world.  It  can  handle  relatively  slowly  evolving  flows  and  observation  platforms  that 
sample  heterogeneously  in  space,  but  assume  that  the  observations  are  taken  at  the  analysis  time. 
However,  highly  intermittent  flows  that  are  not  governed  by  simple  balance  relationships,  and 
observation  systems  that  sample  irregularly  in  time,  or  with  high  temporal  frequency,  are  not  well 
accommodated  within  an  intermittent  3DVAR  framework  but  can  be  accommodated  by  a  4D  data 
assimilation  system.  Furthermore,  an  intermittent  3DVAR  algorithm  produces  a  “snapshot”  of  the 
atmosphere  at  the  center  of  the  typical  6-hour  observation  time  window,  automatically  making  the 
resulting  atmospheric  analysis  at  least  3  hours  old. 

With  NAVDAS-AR,  a  continuous  picture  of  the  atmosphere  over  the  observation  time  window  is 
produced,  providing  an  atmospheric  analysis  at  the  end  of  the  time  window  that  is  current  rather  than  3 
hours  old.  Although  NAVDAS  has  been  quite  successful,  a  4D  data  assimilation  system  is  a  necessity 
to  significantly  improve  not  only  the  accuracy  of  the  common  operational  picture  required  by  the 
warfighter  but  also  the  timeliness  of  providing  this  more  accurate  picture  to  the  warfighter.  The 
advanced  4DVAR  data  assimilation  algorithm,  NAVDAS-AR,  will  provide  the  basis  for  this  system. 
Only  through  4DVAR  algorithms  can  we  truly  exploit  many  of  the  observations  from  current  and 
future  observing  systems.  This  is  especially  important  for  remotely  sensed  observations  that  are 
nonlinearly  and  indirectly  related  to  the  model  state  variables  (e.g.,  satellite  radiances  and  GPS  radio 
occultation  measurements).  In  addition,  the  computational  efficiency  of  NAVDAS-AR  with  respect  to 
the  number  of  observations  makes  it  more  efficient  than  the  NAVDAS  3DVAR  system  in  handling  the 
monumental  increase  in  the  volume  of  satellite  data  expected  over  the  next  decade. 

TRANSITIONS 

Improved  algorithms  for  NAVDAS-AR  have  been  transitioned  to  the  6.4  component  of  this  project, 
and  will  ultimately  be  transitioned  to  FNMOC  in  FY08  as  the  Navy’s  next  generation  operational 
global  atmospheric  data  assimilation  system. 

RELATED  PROJECTS 

Some  of  the  technologies  developed  for  this  project  will  be  used  immediately  to  improve  the  current 
operational  data  assimilation  system  and  the  observational  impact  in  other  NRL  projects.  The 
NAVDAS-AR  has  been  recognized  as  a  viable  framework  for  various  data  assimilation  applications, 
such  as  the  ocean  data  assimilation  system  project  (NRL  base). 
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